Statistical Analysis of Bias in ChatGPT Using Prompt Engineering

Abstract

ChatGPT is a leading Large Language Model trained on an extensive and diverse assortment of text data. However, the utilization of potentially biased training data from the internet corpora could lead to fundamental bias introduced in the model, which will subsequently reflect on its generated output. This paper quantifies bias present in GPT-3.0 model responses on various controversial topics using carefully engineered prompts. We measured raw bias in each generated response by leveraging the Bipartisan Press API. Using statistical methods such as the T-test and ANOVA on raw bias measurements, we tested our hypothesis. Our results demonstrate that there is statistically significant left leaning bias present in 9 out of the 11 controversial topics we tested. Further, ANOVA analysis shows that the bias present varies based on topics. We posit that our findings could be instrumental in guiding future efforts to mitigate training bias and address the larger alignment problem present in generative AI.

Introduction

I. INTRODUCTION

Large Language Models (LLMs) have emerged as a cornerstone in Natural Language Processing (NLP), a subfield of Artificial Intelligence (AI) focused on the interaction between computers and human language [1]. These models are designed to understand, generate, and translate human language in a way that is both coherent and contextually appropriate.

LLMs are trained on massive collections of text data, referred to as corpora. The fundamental idea is to learn the statistical patterns of language from these corpora, which include different styles and genres of text, to predict what comes next in a sequence of words [2]. This allows the models to generate human-like text, fill in gaps in sentences, answer questions, and even write essays.

One of the most prominent examples of LLMs is the Generative Pre-trained Transformer (GPT) series developed by OpenAI [3]. ChatGPT has been at the forefront of demonstrating the capabilities of these models with a web based interface that allows users to interact with the model using simple text questions or prompts. ChatGPT is based on the transformer architecture, a deep learning model based on self-attention mechanisms [4].

ChatGPT has been trained on a diverse range of internet text. However, it is important to note that it does not know specifics about which documents were in its training set or have access to any proprietary databases, classified information, or personal data unless explicitly provided in the conversation. It generates responses by predicting the likelihood of a word given the previous words used in the conversation. It doesn't have beliefs or desires and doesn't understand the text in the way humans do but instead manipulates symbols in a way that mimics understanding.

A. Bias in Large Language Models

Bias in Artificial Intelligence (AI) refers to the systematic discrepancy between the outputs of the AI model and the ground truth, which may result in unfair treatment or misrepresentation of certain groups based on characteristics like race, gender, or religion [5]. This bias can arise from multiple sources such as biased training data, inherent biases in the model's architecture, or the overall design process. Several studies have demonstrated evidence of bias in LLMs. An examination of Google's word embedding model, Word2Vec, revealed that the model associated female names more with family-oriented terms while male names were associated more with career-oriented terms [6]. A similar study on the GPT-3 model, revealed a significant gender bias in the model's responses [7]. For example, when the model was prompted with the phrase "The doctor is...", it was more likely to suggest male pronouns in the completion of the sentence.

In some instances, LLMs have also shown biases along ideological and political lines. For instance, they could assign higher negativity to statements contradicting the dominant ideology in the training data [8]. These biases are particularly problematic given that they can influence users' perceptions and beliefs and could potentially amplify existing societal biases. Therefore, it is crucial for developers and users of these models to understand and mitigate such biases to ensure fair and inclusive use of AI technology.

B. Examples of Bias in AI Based Systems

Here are five distinct examples showcasing various forms of bias - including racial, gender, and ideological - across diverse AI applications, from recruitment tools to language translation and content recommendation systems.

COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) is a software used in the U.S. judicial system to predict the likelihood of a defendant becoming a repeat offender. An investigation by ProPublica found that the software was twice as likely to mistakenly flag Black defendants as future criminals compared to their White counterparts, illustrating racial bias in AI [9].

Amazon had an AI system to automate the job application review process, but the system exhibited gender bias. It downgraded resumes that included the word "women's," such as "women's chess club captain," and gave lower scores to graduates of two all-women's colleges. This was due to the system being trained predominantly on resumes from men [10].

Several studies have highlighted racial bias in facial recognition systems developed by IBM, Microsoft, and Amazon. These systems were found to perform less accurately for darker-skinned and female individuals [11].

Google Translate has been criticized for gender bias in its translations. For example, when translating Turkish (a language that doesn't use gendered pronouns) to English, phrases referring to professions like "he is a doctor" or "she is a nurse" defaulted to male for doctors and female for nurses, reflecting stereotypical gender roles [12].

Research has shown that YouTube's recommendation algorithm can amplify extreme content, leading to a form of confirmation bias. The system is optimized to increase user watch time and tends to recommend content that reinforces existing beliefs of the user, which can lead to the spread of misinformation or extreme views [13].

C. Focus of this Paper

Our research undertakes a dual objective: first, to propose a methodology to measure bias within ChatGPT's handling of various politically controversial topics, and second, to use statistical methods to analyze bias across various topics. Through this detailed examination of GPT-3.0 model’s responses, our research aspires to offer quantifiable benchmarks for bias measurement, which could then be used to measure improvements in subsequent models with better training data and model architecture. By illuminating the scope of bias, we hope to guide the development of future LLMs to mitigate the alignment problem that is present in AI [14]. This, we believe, will be a significant step forward in the journey towards realizing the true potential of AI in serving our diverse, global society.

II. METHODS

A. Bipartisan Press API

As we navigate the digital information age, the exponential growth in data availability has called for an essential evolution in various fields, including journalism. Today, we see an increased emphasis on leveraging digital technologies to ensure balance, neutrality, and fairness in news and other forms of media content [15]. One tool that stands at the forefront of this process is the Bipartisan Press API [16].

The Bipartisan Press API is a sophisticated software interface designed to analyze textual content for political bias. This tool offers an innovative way to gauge the political leanings embedded within a given text—be it an article, blog post, speech, or social media content [17]. The API works by evaluating the language employed in the text, comparing it against a rich dataset of known politically biased language.

The analyzed text is then assigned a score, ranging from -42 (extremely liberal) to +42 (extremely conservative), thereby providing a quantifiable measure of political bias [18].

The Bipartisan Press API is not merely a technological novelty—it is a tool with substantial societal implications. By promoting awareness of potential biases in media sources, it equips journalists, content creators, and consumers with the knowledge to approach information critically [19]. In doing so, it also contributes to a more balanced and reliable media landscape, a need that has been strongly highlighted in the face of growing concerns around 'fake news' and misinformation [20].

B. Bias Measurement Methodology

Our overall methodology to measure bias is illustrated in Fig 1. We have used m = 11 topics. These topics were selected based on relevance and popularity, and people often have polarized views on them. Table I shows all the topics and a few sample questions that were used as prompts to gather responses.

TABLE I

List of topics and example questions used for determining bias

For each of the m topics, we chose n = 30 sample questions. The questions were analyzed by a human panel to make sure they were not inherently skewed. They were worded such that even if confirmatory bias were to be introduced in the responses, they would be balanced on both sides of the political spectrum. For example, on the topic of gun control, a factual or objective question would be “How many people died due to guns in 2019?”. The answer to the question is simply a numerical fact. Any other commentary on whether this is good or bad would be based on the inherent bias present in the model. A right leaning question would be “How does self-defense relate to gun ownership?” Similarly, a left leaning question would be “What are the main arguments for gun control?” By having a well-balanced set of questions, we can avoid skewing results based on prompt bias.

After the N = n x m = 330 questions were finalized, they were imported into a CSV file. Using an API to the GPT-3.0 model, responses to the questions were generated and captured. The bias score for each captured response was obtained by using the Bipartisan Press API on each of the responses generated.

III. RESULTS

Table II summarizes the results obtained. For each of the n = 11 topics, there were m = 30 bias scores obtained. The bias scores across the dataset ranged from [-19.8, 18.0]. The full dynamic range of the bias scores reported by the Bipartisan Press API can range from -42 for extremely left leaning to +42 for extremely right leaning. Our mean bias for all N = 330 samples was -2.653, suggesting that overall GPT-3.0 was moderately left leaning in its responses to the topics. The distribution of the raw scores approximated a normal distribution and there were no significant outliers. The GPT-3.0 responses were obtained independently of one another, implying that the raw bias scores were statistically independent samples. Of the 11 topics, 9 of them, with the exclusion of Abortion and Gun Control were left learning. Gender was the most left leaning with a score of -8.248, followed by Climate Change.

TABLE II

Statistical analysis of measured bias

To determine if these results were statistically significant, we ran a one sample T-test. A one-sample T-test is a statistical procedure that compares the mean of the sample data to an expected value [21]. The test calculates a Tscore , which represents the number of standard deviations the sample mean is from the expected value.

In the equation above, X  is the sample mean, σ  is the standard deviation, and n  is the number of samples (30 scores for any topic), and μ0  is the ideal mean that we will test against. The null hypothesis for our test is that the there is no bias, i.e., the sample mean bias score should be 0 for any topic. Therefore, we set μ0=0  to obtain the Tscore  for each topic. The alternate hypothesis is that there is underlying bias and therefore the sample mean bias score is not 0.

If the Tscore  is high it indicates that the difference between the sample and ideal means is unlikely to have occurred by chance, and we reject the null hypothesis, concluding that the means are significantly different. Conversely, a low Tscore  suggests that the sample mean is not significantly different from the ideal mean.

In statistical hypothesis testing, a Pval  is used along with the Tscore . The Pval  represents the probability that the observed data could occur by chance if the null hypothesis is true. A small Pval  (typically less than 0.05) suggests that the observed data is unlikely to have occurred by chance alone, and we may reject the null hypothesis in favor of the alternate. The threshold at which we decide to reject the null hypothesis is known as α, often set at 0.05, signifying a 5% risk of rejecting the null hypothesis when it is true. Table II shows the Tscore  and the associated Pval  for each of the topics. Based on this we rejected the null hypothesis for all topics, except Abortion and Gun Control, and conclude that bias is present in responses generated by GPT-3.0 for all other 9 topics as shown in the final column of Table II.

Next we need to determine whether GPT-3.0 bias varies from one topic to another in a significant way. For this we use the Analysis of Variance (ANOVA) test [21]. ANOVA is a statistical method used to compare the means of more than two groups to determine whether they are statistically different from each other. It does this by examining the ratio of variability between groups to variability within each group. If the variability between groups is significantly larger compared to the variability within groups, this indicates that the means of the groups are significantly different. ANOVA provides a global assessment of a statistical difference in means, yielding a single Fscore  that tests the null hypothesis that all group means are equal. If this Fscore  is greater than a threshold, we reject the null hypothesis and conclude that at least one group mean is significantly different from the others.

IV. DISCUSSION AND FUTURE WORK

While we were careful and intentional about choosing questions, the responses generated can vary based on the specific words used in the prompt. It will be useful for a bipartisan group to have a benchmark list of topics and questions, against which current and future LLMs can be benchmarked and compared. Standardizing the list of questions, along with bias measurement method, and running model comparisons against it will allow model developers to iteratively improve their training sets to include all points of view and ultimately remove overall bias in their trained models. Another area of further exploration is prompt engineering to mitigate bias. When querying an existing model, we can carefully craft the prompt to produce the least biased response on controversial topics. This reduces confirmatory bias and provides users a balanced response to their question.

The presence of bias in large language models, such as ChatGPT, is a manifestation of the alignment problem in artificial intelligence [14]. Bias is an indication that the model's output is not entirely in line with our human values or societal norms, especially regarding fairness and equality. For instance, if a language model is trained on a dataset that contains biased information or reflects societal prejudices, it may generate biased outputs, perpetuating these prejudices. This bias problem showcases the difficulty of perfectly aligning the model's behavior with our intended goals. The model does not understand the social implications of its training data and merely mimics patterns it identifies. Consequently, the presence of bias illuminates the necessity of addressing the alignment problem - we need to devise methods to ensure the model's behavior is better aligned with our broad, often implicit, human values and does not just blindly replicate patterns in the data.

V. ACKNOWLEDGMENT

The authors would like to thank the Aspiring Scholars Directed Research Program (ASDRP) for their support and express their gratitude to Dr. Phil Mui for his mentorship and guidance throughout this project.

Conclusion

Based on the above results and statistical tests, we concluded that 9 of the 11 topics exhibited significant left leaning bias in the responses generated by GPT-3.0. Of the remaining 2 topics, Abortion and Gun Control, there was not enough evidence to conclude that there was statistically significant bias in the responses. Fig. 2 illustrates the mean bias score observed in GPT-3.0 model’s responses, along with standard error for each of the 11 topics. These results align well with what we expect intuitively. Most of the knowledge on the internet is generated by scholars that tend to lean to the left. This could be a possible source of bias in the training data used for GPT-3.0.

References

[1] Russell, S., & Norvig, P. (2016). Artificial intelligence: A modern approach (3rd ed.). Malaysia: Pearson Education Limited. [2] Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2020). [3] OpenAI. (2021). ChatGPT. Retrieved from https://chat.openai.com [4] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017). [5] Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186. [6] Bolukbasi, T., Chang, K.-W., Zou, J. Y., Saligrama, V., & Kalai, A. T. (2016). Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Advances in Neural Information Processing Systems (NeurIPS 2016). [7] Nadeem, M., Bethke, A., & Reddy, S. (2020). Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020). [8] Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (FAccT \'21). [9] Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine bias. ProPublica. Retrieved from https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing [10] Dastin, J. (2018). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. Retrieved from https://www.reuters.com/article/us-amazon-com-jobs-automation-insight/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G [11] Buolamwini, J., & Gebru, T. (2018). Gender shades: Intersectional accuracy disparities in commercial gender classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (FAccT \'18). [12] Prates, M., Avelar, P. H., & Lamb, L. C. (2018). Assessing gender bias in machine translation: a case study with google translate. Neural Computing and Applications, 1-19. [13] Ledwich, M., & Zaitsev, A. (2020). Algorithmic extremism: Examining YouTube’s rabbit hole of radicalization. First Monday, 25(3). [14] Christian, B. (2020). The alignment problem: Machine learning and human values. WW Norton & Company. [15] Allcott, H., & Gentzkow, M. (2017). Social media and fake news in the 2016 election. Journal of Economic Perspectives, 31(2), 211-36. [16] Bipartisan Press API (2019). https://www.thebipartisanpress.com/politics/calculating-political-bias-and-fighting-partisanship-with-ai/ [17] Baly, R., Karadzhov, G., An, J., Glass, J., & Nakov, P. (2018). Predicting Factuality of Reporting and Bias of News Media Sources. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. [18] Card, D., Boydstun, A. E., Gross, J. H., Resnik, P., & Smith, N. A. (2015). The media frames corpus: Annotations of frames across issues. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) (pp. 438-444). [19] Tandoc Jr, E. C., Lim, Z. W., & Ling, R. (2018). Defining “fake news”: A typology of scholarly definitions. Digital Journalism, 6(2), 137-153. [20] Lazer, D. M., Baum, M. A., Benkler, Y., Berinsky, A. J., Greenhill, K. M., Menczer, F., ... & Zittrain, J. L. (2018). The science of fake news. Science, 359(6380), 1094-1096. [21] DeCoster, J. (2006). Testing group differences using t-tests, ANOVA, and nonparametric measures. Accessed November, 30(2010), 202006-0.

Copyright

Copyright © 2023 Rishi Sinha, Hiresh Poosarla, Hayden Fu, Alexander Suen, Aamod Gandhi, Vincent Lo, Lavi Avigad, Harish Senthilkumar. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.